1. Introduction
The optical microscope in the diagnostic and biomedical laboratory is routinely used by pathologists and research scientist to make diagnosis and perform experiments. These users perform these functions by visualizing cells and tissue sections that have been previously prepared and chemically stained in the histology or histochemistry laboratory. Every patient with a tumor suspected of cancer undergoes evaluation with the most critical pathway involving a tissue biopsy. The biopsy tissue is routinely fixed in formalin, processed in a tissue processor, embedded in formalin and serially cut in a microtome to give thin sections representing the diagnostic material. The diagnostic material then is a representative tissue section with tangentially cut whole cells and chemically marked with mordant dyes and indicia markers. One of the ubiquitous dyes is the nuclear counter stain hematoxylin and one of the common indicia markers are the monoclonal antibody or nuclear acid probes tagged with an enzyme reagent and a chromogenic substrate. The most common chromogenic substrate is DAB (diaminobenzidine) which is visualized as reddish brown and the most common nuclear counter stain is hematoxylin which is visualized as blue.
The diagnosis is performed by examining the tissue optically using the objective lenses of the microscope in low and high power magnifications. The routinely stained hematoxylin and eosin tissue is examined first to visualize the presence of tumor or benign cells and in the majority of cases, confirmed by a panel or set of immunohistochemical stains targeting lineage, proliferative, tumor associated or prognostic or oncogenic markers. The current state of the art of diagnosis is to estimate the percentage of immunohistochemically stained cells and based on this subjective interpretation render a diagnosis. No tool is currently available to use computerized image analysis to count and display these relevant cells for the pathologist or scientist. Counting and identifying these cells are crucial in making decisions for diagnosis or prognosis, yet the diagnostic practice relies on a subjective approach, even though patient outcomes and treatment decisions are at stake.
The latter practice is the standard of practice, not because it is the optimal way, but because of an absence of an automated cell-quantifying instrument attached to the microscope. This practice is subjective, error prone, and often gives wide range of results that depends on the level of microscopist's skill. This is due to difficulty in counting positive cells accurately because of overlapped stained nuclei, variability of staining, and the limitation of our visual system.
To analyze immunostained cells, we have two major techniques: flow cytometry and immunohistochemistry. On one hand, the flow cytometer, requires a viable tissue disaggregated to individual live cells to quantify the cells. These viable cells are identified using fluorescent-tagged antibody probes in a highly accurate way, but will not allow concurrent visualization of the cells analyzed. Immunohistochemistry, on the other hand, uses formalin-fixed non-viable tissue specimen and chromogen-tagged antibodies of defined specificity to identify, mark, and concurrently visualize specific types of cells, the latter function not present in flow cytometry. There is desire and need for the pathologists to both quantify and see tissues to have valid, real time, objective feedback on the types and cells identified to make the crucial diagnosis or prognosis.
However, there is currently no system that will perform “flow cytometry” to identify the types and the percentage of the immunostained cells in fixed tissue. Using our novel proposed technology, we combine the advantages provided by flow cytometry in quantifying cells and also retain the advantages of microscopy in morphologically visualizing the immunoreactive cells. To accomplish this aim, we resort to new and improved advance image analysis using a surprisingly easy and unique, useful, novel process as described herein.
Immunohistochemistry (IHC) is indispensable in clinical practice yet a tool to count cells in a novel intuitive way is not available and is needed. The current state of the clinical art in pathology diagnosis allows the pathologists to either make a judgment call for a positive or negative result of immunohistochemistry or semi quantitatively grade the percentage of relevant positive or negative population and give a percentage estimate based on the pathologist subjective feel of the extent of positive reaction. Routinely in pathology practice, a panel of 5 to 15 immunohistochemistry antibodies are applied on the slide-based tissue sections to create a differential matrix to rule in or out a diagnosis based on the tumor associated markers. Most of diagnostic pathology, whether a small office or a large reference laboratory, uses immunohistochemistry as part of a standard of practice. In practice, the use of IHC may shift the diagnostic probability, for example from 75% to 100%. This is especially true in hematopathology diagnosis where an enhanced diagnostic accuracy is reported if the immunologic results are included (Blood, Armitage et al., Int Lymphoma Study Group, 1997). The enhanced accuracy is reported to an increased accuracy beyond the routine hematoxylin and eosin stained tumor from 5 to 35% of the cases.
Current image analysis in diagnostic centers are specialized tools to semiquantitate hormone receptor antigen for prognosis only. Yet none of these diagnostic centers have an automated method with which to count cells in immunohistochemistry stained slides in other tumor types or even in cancers of the lymphatics such as lymphomas. Automated detection of chromogen stained biological cells in tissue in a population statistic manner has lagged behind quantitation of antigen in tissue and cells for prognosis and diagnosis, i.e., Her2neu, ER, PR hormones profile for breast cancer.
Current image analysis approaches and those systems describe above are inadequate to perform a “virtual flow cytometry” on tissue. Many of the tissues submitted for diagnosis are fixed in formalin and subjected to immunohistochemistry to aid or confirm the diagnosis. In immunohistochemically stained cells in tissue, the cells are often ambiguously and syncitially linked, variable in size, variable in intensity staining, variable in color staining, with much overlap that even expert guided manual counting is difficult to be accurate. Despite these obvious difficulties, the percentage of positive staining cells is currently estimated visually by eye without the aid of a computerized tool. The level of accuracy of expert observers varies by as much as 25%. The goal then is to exceed this performance using rapid and robust computerized automation.
Current image analysis techniques perform image analysis based on chromogen associated pixel comparison using a dedicated instrument with transmitted light operation set within a narrow range. The problem with this prevailing approach is that the chromogen associated pixels often are associated with the pixels with the counter stain dyes. A cell has a nucleus, a cytoplasm and a surface cell membrane. Membrane reactive brown chromogen bleeds into the cytoplasm and include most of the nuclear area as well (FIG. 3, color frame 14 and FIG. 4a in drawings). These color mixture makes it difficult to isolate the brown only pixels and simple detection of antigen density by looking for brown pixels will not be able to easily extract the brown chromogen apart from the blue dye. Moreover, the staining variability and tumor antigen expression variability may increase or decrease color intensity of these chromogen. This variability is not so easily correlated with pixel distribution. The staining variability is also related to the level of transmitted light. By limiting this variable, a pathologist who often obtain images from microscope with little regard for a set light but based on comfort of vision, often extract images in random light intensity. The prior art limitation by being a dedicated machine with set lighting precludes routine use in diagnostic pathology and evaluation of immunohistochemistry in a routine manner.
Because these approaches do not detect single cells of the same type or class, no single cell percentage could be obtained. The usual result is percent of pixels overall the area examined. These areas are often called hot spots to indicate an approximate location of relevant cells.
Segmentation of biological images of chromogen-marked microscopic cellular images is difficult because of the variability of these images. Color in immunostained cells in tissue varies from strongly stained to weakly stained cells. The chromogens used may also vary. Furthermore, color segmentation tools are not readily available or easily applied. RGB (Red Green Blue) by itself, its various expressions and combinations as used in many current systems and approaches, are tightly linked with intensity component and therefore, any ratio derived from them will be biased by the black and white components of the image. True color image analysis is not achievable. Therefore, these algorithms rely on grayscale-discriminating segmentation paradigms which are incapable of solving problems of variability in staining, or the identification of nuclei of unstained and stained cells, and cells stained with different color of chromogens (tissue stain). The difficulty lies in the inability of these paradigms to separate intensity from chromatic properties of tissue stains.
There is still a need to bridge the perception prevalent in literature on microscopic images and the low-level image features that most algorithm are based on. Current algorithms try to find the best technique to solve technical problems on limited data sets addressing solution to historical problems by solving to the level of the primitives and comparing results with other approaches. One difficulty of this approach in the real world problems in biological detection is complex and that most often, it is the combination of techniques and the empirical adaptive human responses to the results that point to the acceptable solutions. The ground truth in most biological images in the domain of automated immunohistochemistry may be fuzzy, ill defined and subjective. Therefore, it is not so much as the accuracy of thresholding the exact boundaries of individual object that may be relevant but it is as much as the relevance of enumerating the individual objects of the population being studied. It is like the problem of hitting the bull but not necessarily the bull's eye. It also follows that the approach to solve this generic problem is not to develop low-level feature detection algorithms but on the development of a combination of low-level features detection tempered by the feedback from human observers.
2. Prior Art
U.S. Pat. No. 6,692,952 Feb. 17, 2004 Braff, R. MIT
This invention relates to cell analysis and sorting devices and methods for manipulating single cells using these microscopic devices. The devices use cells in fluidics similar to flow cytometry and does not use routine stained slides by immunohistochemistry means.
U.S. Pat. No. 6,294,331 Sep. 25, 2001 Ried, T. USA
This invention relates to methods of detecting genetic and phenotypic markers in biological samples on slides using spectral imaging and brightfield microscopy to detect the presence of chromogenic dyes. The analysis is not single cell and will not perform percentage of the same class of cells.
U.S. Pat. No. 6,215,892 2001-2004 Douglass, J. Chromavision
The present invention has utility in the field of oncology for the early detection of minimal residual disease (“micro metastases”) on microscopic slides but does not seek or report the percentage of single cells.
U.S. Pat. No. 6,418,236 2002-2007 Ellis, B. Chromavision
The invention relates generally to light microscopy and, more particularly, to automated techniques of analyzing cytochemical and immunohistochemical staining on slides. The method and results, though based on color ratios of RGB, are not based on single cell analysis of same class of cells and will not present results in a two dimensional histogram.
U.S. Pat. No. 6,404,916 Jun. 11,2002 De La Torre-Bueno, J. Chromavision
This invention deals with an apparatus of digital components to perform color threshold analysis by volume distribution. The subject is locally adaptable in machine vision field and may not be useful in detecting immunostained cells in tissue, wherein these cells are in a contiguous distribution with a gradation and mixture of bleeding colors, as is often the situation with immunohistochemical stains of cells using brown chromogen and blue counterstain. The periphery of the cell is brown and the center is blue precluding use of a color analyzer predicated on homogeneous color volumes. This invention takes teaching from one well known classical method that converts the RGB color information into another color space, such as HSI (hue, saturation, intensity) space (1) Two book references by Russ J C, and (2) Gonzales R C addressed this issue in detail. In such a space, distinctly different hues such as red, blue, green, yellow, may be readily separated. In addition, relatively lightly stained objects may be distinguished from more intensely stained ones by virtue of differing saturations. Converting from RGB space to HSI space requires more complex computation not necessarily needing a dedicated hardware as this invention is about, but is within the real time span methods of the current image processors and personal computers with fast central processors.
U.S. Pat. No. 6,337,472 Jan. 8, 2002 Garner, H. Univ. of Texas
The present invention relates in general to the field of biological sample analysis, and more particularly, to an apparatus and method for observing, identifying and quantifying a biological sample through a microscope using the entire spectrum of light, concurrently and in real time. There is not single cell identification but the invention is predicated on pixel distribution of detected moieties.
Additional Commercial Products:
The Compucyte's laser scanning cytometer technology grew out of the original high-content cell analysis technology: Flow Cytometry by using fluorescence and laser light scattering methods, and then analyzing that data with powerful graphical software tools to obtain meaningful population-based information. The system will not perform chromogen based brightfield cell analysis. The newer system called iColor will perform cell analysis using combined fluorescence and chromogen stain but is still a cell based system using segregated cells in a proprietary cell well substrate. It does not perform on a regular tissue immunohistochemistry stained slide which is the current state of art in pathology practice.
The Chromavision ACIS, with some of their patents described above, could do many slide based analysis but has limited the population statistic analysis to getting pixels that are positively stained in hot spots areas and over all the area of the image frame. It uses an RGB color ratio and color transform as well as lookup table and work with single pixels, not single cell analysis. The percent obtained in their instrument relate to percent area of the image.
Since the invention may be seen as similar to Chromavision, ACIS, we extract the relevant article that separates our invention. We do not use RIB ratio but uses a different method of extracting separately the blue and red thresholded objects, work on this local regions of interest and not on the total frame, and perform a dynamic color and intensity segmentation on these thresholded bitplane binary objects. Furthermore, it is clear that their technique is an estimation based on the area and the average size of cell nuclei, which clearly departs from our single cell technique. Their R/B ratio technique is stated herein for reference:
“Thus, the pixels of a cell of interest which has been stained contain a red component which is larger than either the green or blue components. A ratio of red divided by blue (R/B) provides a value which is greater than one for tumor cells but is approximately one for any clear or white areas on the slide. Since the remaining cells, i.e., normal cells, typically are stained blue, the R/B ratio for pixels of these latter cells yields values of less than one. The R/B ratio is preferred for clearly separating the color information typical in these applications. Since it is of interest to separate the red stained tumor cells from blue stained normal ones, the ratio of color values is then scaled by a user specified factor. As an example, for a factor of 128 and the ratio of (red pixel value)/(blue pixel value), clear areas on the slide would have a ratio of 1 scaled by 128 for a final X value of 128. Pixels which lie in red stained tumor cells would have X value greater than 128, while blue stained nuclei of normal cells would have value less than 128. In this way, the desired objects of interest can be numerically discriminated. It has been found that normal cells whose nuclei have been stained with hematoxylin are often quite numerous, numbering in the thousands per 10. times. image. Since these cells are so numerous, and since they tend to clump, counting each individual nucleated cell would add an excessive processing burden, at the expense of speed, and would not necessarily provide an accurate count due to clumping. The apparatus performs an estimation process in which the total area of each field that is stained hematoxylin blue is measured and this area is divided by the average size of a nucleated cell. By dividing this value by the average area for a nucleated cell at 350, and looping over all fields at 352, an approximate cell count is obtained. Preliminary testing of this process indicates an accuracy with +/−15%.”
The ARIOL system of Applied Imaging uses automated slide delivery to microscope and performs similar capacities as the Chromavision. It has been using the pixel mask technology and because of similar accuracy issues has not implemented its population statistic reporting.
PAXIT has a limited module to do population statistics but only appears to count the positively stained cells in a nuclear pattern, not in a single cell mode.
Imaging flow cytometry U.S. Pat. No. 6,251,615 will not allow visualization of routinely immunohistochemically stained cells in brightfield microscopy but uses fluorochrome reactive antigens and fluorescent microscopy displayed cells. An example is ImageStream® 100 Imaging Flow Cytometer which is high-throughput system (200 cells/second) that generates brightfield, darkfield and up to four fluorescent images, but will not perform single cell analysis on routinely immunohistochemically stained slides.
It has been found, however, that present prior art apparatus and methods fail to meet the demand for a low cost, efficient, customizable imaging microscope that is capable of extracting or overlapping, concurrent data acquisition and analysis over color image obtained by brightfield light. A problem found in alternative systems is that they are capable of imaging a set of pixels representing the stained object over all the other digital objects in the image frame which is not an accurate representation since some microscopic images contain background stromal tissue or other cells other than the relevant class. Examples of these other objects include stroma, blood vessels, large cancer cells if the target cells are the tumor reactive lymphocytes, fat and serum protein spaces. Another problem with available systems is the need for special filters, reliance on machine obtained hot spots and non-biased approaches, and complexity in the system optics is required, increasing the complexity to user and system.